Method and apparatus for generating an artificial intelligence 3D dataset and performing interactive manipulation and rendering of the dataset

ABSTRACT

A method comprises generating a 3D volumetric dataset through an artificial intelligence process. Then, performing a simulation by first assigning mechanical type properties to a 3D volumetric dataset. Then, performing rendering of the 3D volumetric wherein the 3D volumetric dataset has a first configuration. Then, receiving an input to cause the 3D volumetric dataset to change from a first configuration to a second configuration wherein the change in configuration is in accordance with the nature of the input and the mechanical type properties of the 3D dataset. Then, performing rendering of the 3D volumetric dataset in the second configuration. This cycle is repeated over multiple changes in configuration.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation-in-part of application Ser.No. 16/195,251, filed on Nov. 19, 2018, which is a continuation-in partof application Ser. No. 15/904,092, filed on Feb. 23, 2018. Provisionalapplication No. 62/695,868, filed on Jul. 10, 2018, provisionalapplication No. 62/651,934, filed on Apr. 3, 2018, provisionalapplication No. 62/628,527, filed on Feb. 9, 2018.

TECHNICAL FIELD

Aspects of this disclosure are generally related to radiologicalimaging, and more particularly to surgical planning.

BACKGROUND

One of the challenges that interventional radiologists and surgeons faceprior to performing a surgery is selection of optimal hardware. Becauseof uncertainties, the surgeon may select several slightly differentpieces of hardware for the same purpose, of which one piece may beoptimal. Which piece of hardware is optimally suited to the purpose isdetermined during surgery.

SUMMARY

All examples, aspects and features mentioned in this document can becombined in any technically possible way.

In accordance with an aspect a method comprises: assigning tissue typeproperties to voxels of a medical image; and performing athree-dimensional simulation by manipulating the voxels based on theassigned tissue type properties and an input that prompts voxelmanipulation. In some implementations assigning tissue type propertiesto the voxels of the medical image comprises assigning a value for atleast one of: elasticity, ductility, hardness, density, and thermalconductivity. In some implementations manipulating the voxels compriseschanging at least one of voxel size, voxel location, voxel orientation,voxel shape, voxel color, voxel grayscale, and voxel tissue typeproperty value. In some implementations the input comprises inserting avirtual volume-subtending surgical object. In some implementations theinput comprises inserting a virtual volume-subtending anatomic object.In some implementations performing the simulation comprises representingvirtual motion. In some implementations performing the simulationcomprises representing deformation of tissue. In some implementationsperforming the simulation comprises representing a radiologicaldissection. In some implementations performing the simulation comprisescreating new voxels in response to the input. In some implementationscreating new voxels comprises creating new fixed-type voxels that do notchange unless acted upon by a user through an interface. In someimplementations creating new voxels comprises creating newinvisible-type voxels. In some implementations creating new voxelscomprises creating new tissue-type voxels. In some implementationscreating new voxels comprises creating new dynamic-type voxels having atleast one tissue type property that changes over time. In someimplementations creating new voxels comprises creating new mobile voxelsthat travel through a virtual vascular structure. In someimplementations performing the simulation comprises eliminating selectedvoxels in response to the input. Some implementations compriseeliminating tissue type voxels to simulate ablation. Someimplementations comprise inserting strategic elimination points thatdesignate a direction in which voxels are eliminated in discrete steps.Some implementations comprise inserting strategic non-elimination pointsthat designate voxels that are preserved from elimination. Someimplementations comprise assignment of a strategic deformation featurethat designates a maximum shift for at least one voxel.

In accordance with an aspect an apparatus comprises: an IO device; andan image processor in communication with the IO device, the imageprocessors comprising a program stored on computer-readablenon-transitory media, the program comprising: instructions that assigntissue type properties to voxels of a medical image; and instructionsthat perform a three-dimensional simulation by manipulating the voxelsbased on the assigned tissue type properties and an input that promptsvoxel manipulation. In some implementations the instructions that assigntissue type properties to voxels assign a value for at least one of:elasticity, ductility, hardness, density, and thermal conductivity. Insome implementations the instructions that perform the three-dimensionalsimulation by manipulating the voxels change at least one of voxel size,voxel location, voxel orientation, voxel shape, voxel color, voxelgrayscale, and voxel tissue type property value. In some implementationsthe input comprises insertion of a virtual volume-subtending surgicalobject. In some implementations the input comprises insertion of avirtual volume-subtending anatomic object. In some implementations thesimulation comprises representing virtual motion. In someimplementations the simulation comprises representing deformation oftissue. In some implementations the simulation comprises representing aradiological dissection. In some implementations the simulationcomprises creation of new voxels in response to the input. In someimplementations creation of new voxels comprises creation of newfixed-type voxels that do not change unless acted upon by a user throughan interface. In some implementations creation of new voxels comprisescreation of new invisible-type voxels. In some implementations creationof new voxels comprises creation of new tissue-type voxels. In someimplementations creation of new voxels comprises creation of newdynamic-type voxels having at least one tissue type property thatchanges over time. In some implementations creation of new voxelscomprises creation of new mobile voxels that travel through a virtualvascular structure. In some implementations the simulation compriseselimination of selected voxels in response to the input. In someimplementations elimination of selected voxels comprises elimination oftissue type voxels to simulate ablation. Some implementations comprisestrategic elimination points that designate a direction in which voxelsare eliminated in discrete steps. Some implementations comprisestrategic non-elimination points that designate voxels that arepreserved from elimination. Some implementations comprise a strategicdeformation feature that designates a maximum shift for at least onevoxel.

In accordance with an aspect, a method comprises: using a 3D volumetricmedical imaging dataset; performing anatomic segmentation of voxels inthe volumetric medical imaging dataset into distinct tissue types;assigning tissue properties (mechanical properties such as elasticity,ductility, hardness or physical properties such as density or thermalconductivity) to each tissue type in the medical imaging dataset by acombination of properties acquired from the scanner and by lookup table;performing options for voxel manipulation (e.g., changing at least onevoxel's size, location, orientation, shape, color/grayscale, or othervoxel property); performing options for voxel creation (e.g.,fixed-location voxel creation such as tissue-type or invisible-typevoxel creation, dynamic voxel creation such as virtual contrast, orinteractive voxel creation, such as a virtual occluder that can interactwith other voxels in the 3D medical imaging); performing options forvoxel elimination (e.g., eliminate and discard via volume-typeelimination or surface-type elimination, cut and place voxels into avirtual specimen container bucket; perform new applications includingvirtual motion, deformable tissue, virtual radiological dissection,perform annotations or recording the interactive steps for replay.

In accordance with an aspect a method comprises: performing ofmanipulation of at least one of the original voxel parameters including,but not limited to, at least one of the following: change size (e.g.,increase or decrease at least one dimension of at least one voxel);change shape (e.g., change from cube shaped voxel to cylindrical shapedvoxel); change in position (i.e., the center coordinate of the voxelchanges in the x-direction, y-direction, and/or z-direction); change inorientation (i.e., the orientation of the voxel changes in roll, pitchand/or yaw); or change in internal parameter (e.g., texture, tissueproperty, etc.). Thus, at least one voxel has a final voxel parameterthat differs from the original voxel parameter.

In accordance with an aspect a method comprises: performing anatomicsegmentation of voxels in the 3D imaging dataset, such that all voxelsare assigned both a tissue type (e.g., skin, fat, bone, cerebrospinalfluid, etc.) and tissue properties (e.g., mechanical properties such aselasticity, ductility, hardness, physical properties such as density orthermal conductivity, etc.) by a combination of properties acquired fromthe scanner, tissue segmentation and by lookup table.

In accordance with an aspect a method comprises: creation of fixed-typevoxels. This implementation includes the creation of invisible-typevoxels in between two closely spaced structures, such that thestructures can be separated from one another. Another implementation isthe creation of tissue-type voxels to stretch a structure. Anotherimplementation is the creation of a combination of invisible-type voxelsand tissue-type voxels, such that structures can be pulled apart for thepurposes of untangling and better visualization of complex structures(e.g., improve visualization of a cerebral arteriovenous malformation).Other fixed-type voxels include, but are not limited to, the following:surgical-device-type voxels; fluid-type voxels (e.g., water, blood,etc.); non-fluid-type voxels (e.g., kidney tissue, etc.); or, others.This implementation includes creation of annotation-type voxels, whichare attached to the structure of interest, but serve to facilitateunderstanding or communication of a finding. An example is annotation ofvoxel counting metrics, such as the markup of curvilinear distance. Notethat when viewing with 3D and rotation is used, the annotations willrotate such that they are optimally viewed from each viewingperspective.

In accordance with an aspect a method comprises: creation ofdynamic-type voxels. This implementation includes the creation of mobilevoxels, which have the capability to move through tubular shapedvascular structures from one end to another (e.g., proximal to distal)to improve visualization. The manner in which the mobile voxels movethrough the tubular vascular structures can be in accordance with anyflow patterns, such as laminar flow or plug flow.

In accordance with an aspect a method comprises: creation ofinteractive-type voxels. This implementation includes the creation ofvoxels, which have the capability to interact with any voxels or tissueswithin the dataset, such as placement of voxels to mimic a surgical clipand occlude the flow of virtual contrast along some portion of a tubularblood vessel. Another example is soft tissue voxels changing in relationto the placement of a virtual occluder type interactive voxels. Anotherexample of an interactive-type voxel is the insertion of a strategicdeformation voxel, which can be used to guide the deformation of softtissue voxels when a virtual surgical object is being inserted.

In accordance with an aspect a method comprises elimination of voxels ineither a eliminate and discard approach (e.g., volumetric-typeelimination, multi-step layer-by-layer ablative-type approach orelimination associated with the placement of a virtual object, cardinaldirection-type elimination). One element to aid in the elimination ofthe desired voxels is through the use of strategic elimination pointsand strategic non-elimination points. Some implementations use a 3Dcursor to eliminate voxels outside of the cursor. Some implementationsuse an ablative approach to take the form of elimination of the outervoxels from the whole surface, one voxel layer at a time. Otherimplementations with an ablative approach can take the form ofelimination of the outer voxels from a portion of the whole surface, onevoxel layer at a time. Other types include the elimination in accordancewith placement of a virtual object.

In accordance with an aspect a method comprises: performing acoordinated multi-voxel manipulation to simulate motion of anatomicalstructures in accordance with the intrinsic properties (e.g., hardness,elasticity, etc.) of the voxels composing the anatomic structure. Twoexamples were provided in this patent for illustrative purposes. Thefirst example is virtual motion at the knee joint. The second example isthe movement of a virtual catheter inside a blood vessel, with anadditional viewing option to include a tunnel view (as if the viewpointwas just short of the catheter tip and viewing both the catheter tip andthe branches.

In accordance with an aspect, a method comprises: assignment ofstrategic deformation points (e.g., the user can specify which voxelsare more or less deformable than predicted by the algorithm based onhis/her prior knowledge to guide the coordinated multi-voxel shift toachieve the desired deformation); determination of whether insertion ofthe surgical object is possible based on geometric fitting of an objectin accordance with compliance of adjacent tissues, which if possible,the 3D digital object is placed along with a coordinated multi-voxeltissue shift, and which if not possible, an algorithm to inform the userfeedback as to why the 3D digital object will not fit along withopportunities for additional attempts for insertion of the 3D digitalobject or insertion of the 3D digital object with a combination of somenative voxels replaced by the 3D digital object and some native voxelsshifted or perform insertion of the 3D digital object with purereplacement of native voxels with voxels corresponding to the 3D digitalobject. Examples provided in this patent with at least some degree ofdeformation of native tissues include the following: insertion of a 3Ddigital representation of a breast implant with variable deformation ofthe breast; insertion of a 3D digital representation of a nasal implantwith variable deformation of the nasal tissues over time; insertion of a3D digital representation of a renal implant from a donor kidney withdeformation of the adjacent adrenal gland; and, insertion of a breastmass at one time point into the same voxel space of the breast mass at asecond time point with special deformation of the one of the breastmasses to provide optimum comparison of how the mass changes over time.Examples provided in this patent where there is no deformation oftissues and only replacement of voxels include the following: insertionof a 3D digital object representing a radiofrequency ablation zone;insertion of a virtual coil into an aneurysm and insertion of femoralneck fixation hardware into the femur.

In accordance with an aspect a method comprises: performingfiltering/segmentation, using a 3D cursor, 3D viewing via augmentedreality/virtual reality headset, manipulation of voxel(s), creation ofvoxel(s) and elimination of voxel(s) to optimize viewing of a structure.The example provided in this patent is of a cerebral arteriovenousmalformation (AVM). In this example, the feeding arteries, AVM nidus anddraining veins will be labeled. Any structures that block the viewing ofthe AVM will be either removed or made less conspicuous. The tightlypacked blood vessels will be spread apart through insertion of invisiblevoxels and tissue-type voxels. Through coordinated multi-voxelmanipulations, voxel creation and voxel elimination, the AVM can beuntangled (e.g., straightened, stretched, bent, twisted, expanded,contracted, cut, etc.) and the connections and AVM anatomy betterunderstood. Through this virtual radiological dissection and through theinsertion of dynamic voxels in the form of virtual contrast into an AVMaccompanied by varying virtual occluder insertion patterns, theunderstanding of the complex AVM anatomy can be enhanced in anon-invasive manner.

In accordance with an aspect an apparatus comprises: an input/output(TO) device; and an image processor in communication with the IO device,the image processors comprising a program stored on computer-readablenon-transitory media, the program comprising: instructions use a 3Dvolumetric medical imaging dataset and perform anatomic segmentation ofvoxels in the volumetric medical imaging dataset into distinct tissuetypes; assign tissue properties (mechanical properties such aselasticity, ductility, hardness or physical properties such as densityor thermal conductivity) to each tissue type in the medical imagingdataset by a combination of properties acquired from the scanner and bylookup table; perform options for voxel manipulation (e.g., changing atleast one voxel's size, location, orientation, shape, color/grayscale,or other voxel property); perform options for voxel creation (e.g.,fixed-location voxel creation such as tissue-type or invisible-typevoxel creation, dynamic voxel creation such as virtual contrast, orinteractive voxel creation, such as a virtual occluder that can interactwith other voxels in the 3D medical imaging); perform options for voxelelimination (e.g., eliminate and discard via volume-type elimination orsurface-type elimination, cut and place voxels into a virtual specimencontainer bucket; perform new applications including virtual motion,deformable tissue, virtual radiological dissection, perform annotationsor recording the interactive steps for replay.

In accordance with an aspect an apparatus comprises: performing ofmanipulation of at least one of the original voxel parameters including,but not limited to, at least one of the following: change size (e.g.,increase or decrease at least one dimension of at least one voxel);change shape (e.g., change from cube shaped voxel to cylindrical shapedvoxel); change in position (i.e., the center coordinate of the voxelchanges in the x-direction, y-direction, and/or z-direction); change inorientation (i.e., the orientation of the voxel changes in roll, pitchand/or yaw); or change in internal parameter (e.g., texture, tissueproperty, etc.). Thus, at least one voxel has a final voxel parameterthat differs from the original voxel parameter.

In accordance with an aspect an apparatus comprises: performing anatomicsegmentation of voxels in the 3D imaging dataset, such that all voxelsare assigned both a tissue type (e.g., skin, fat, bone, cerebrospinalfluid, etc.) and tissue properties (e.g., mechanical properties such aselasticity, ductility, hardness, physical properties such as density orthermal conductivity, etc.) by a combination of properties acquired fromthe scanner, tissue segmentation and by lookup table.

In accordance with an aspect an apparatus compromises: creation offixed-type voxels. This implementation includes the creation ofinvisible-type voxels in between two closely spaced structures, suchthat the structures can be separated from one another. Anotherimplementation is the creation of tissue-type voxels to stretch astructure. Another implementation is the creation of a combination ofinvisible-type voxels and tissue-type voxels, such that structures canbe pulled apart for the purposes of untangling and better visualizationof complex structures (e.g., improve visualization of a cerebralarteriovenous malformation). Other fixed-type voxels include, but arenot limited to, the following: surgical-device-type voxels; fluid-typevoxels (e.g., water, blood, etc.); non-fluid-type voxels (e.g., kidneytissue, etc.); or, others. This implementation includes creation ofannotation-type voxels, which are attached to the structure of interest,but serve to facilitate understanding or communication of a finding. Anexample is annotation of voxel counting metrics, such as the markup ofcurvilinear distance. Note that when viewing with 3D and rotation isused, the annotations will rotate such that they are optimally viewedfrom each viewing perspective.

In accordance with an aspect an apparatus comprises: creation ofdynamic-type voxels. This implementation includes the creation of mobilevoxels, which have the capability to move through tubular shapedvascular structures from one end to another (e.g., proximal to distal)to improve visualization. The manner in which the mobile voxels movethrough the tubular vascular structures can be in accordance with anyflow patterns, such as laminar flow or plug flow.

In accordance with an aspect an apparatus comprises: creation ofinteractive-type voxels. This implementation includes the creation ofvoxels, which have the capability to interact with any voxels or tissueswithin the dataset, such as placement of voxels to mimic a surgical clipand occlude the flow of virtual contrast along some portion of a tubularblood vessel. Another example is soft tissue voxels changing in relationto the placement of a virtual occluder type interactive voxels. Anotherexample of an interactive-type voxel is the insertion of a strategicdeformation voxel, which can be used to guide the deformation of softtissue voxels when a virtual surgical object is being inserted.

In accordance with an aspect an apparatus comprises elimination ofvoxels in either a eliminate and discard approach (e.g., volumetric-typeelimination, multi-step layer-by-layer ablative-type approach orelimination associated with the placement of a virtual object, cardinaldirection-type elimination). One element to aid in the elimination ofthe desired voxels is through the use of strategic elimination pointsand strategic non-elimination points. Some implementation use a 3Dcursor to eliminate voxels outside of the cursor. Some implementationsuse an ablative approach to take the form of elimination of the outervoxels from the whole surface, one voxel layer at a time. Otherimplementations with an ablative approach can take the form ofelimination of the outer voxels from a portion of the whole surface, onevoxel layer at a time. Other types include the elimination in accordancewith placement of a virtual object.

In accordance with an aspect an apparatus comprises: performing acoordinated multi-voxel manipulation to simulate motion of anatomicalstructures in accordance with the intrinsic properties (e.g., hardness,elasticity, etc.) of the voxels composing the anatomic structure. Twoexamples were provided in this patent for illustrative purposes. Thefirst example is virtual motion at the knee joint. The second example isthe movement of a virtual catheter inside a blood vessel, with anadditional viewing option to include a tunnel view (as if the viewpointwas just short of the catheter tip and viewing both the catheter tip andthe branches.

In accordance with an aspect, an apparatus comprises: assignment ofstrategic deformation points (e.g., the user can specify which voxelsare more or less deformable than predicted by the algorithm based onhis/her prior knowledge to guide the coordinated multi-voxel shift toachieve the desired deformation); determination of whether insertion ofthe surgical object is possible based on geometric fitting of an objectin accordance with compliance of adjacent tissues, which if possible,the 3D digital object is placed along with a coordinated multi-voxeltissue shift, and which if not possible, an algorithm to inform the userfeedback as to why the 3D digital object will not fit along withopportunities for additional attempts for insertion of the 3D digitalobject or insertion of the 3D digital object with a combination of somenative voxels replaced by the 3D digital object and some native voxelsshifted or perform insertion of the 3D digital object with purereplacement of native voxels with voxels corresponding to the 3D digitalobject. Examples provided in this patent with at least some degree ofdeformation of native tissues include the following: insertion of a 3Ddigital representation of a breast implant with variable deformation ofthe breast; insertion of a 3D digital representation of a nasal implantwith variable deformation of the nasal tissues over time; insertion of a3D digital representation of a renal implant from a donor kidney withdeformation of the adjacent adrenal gland; and, insertion of a breastmass at one time point into the same voxel space of the breast mass at asecond time point with special deformation of the one of the breastmasses to provide optimum comparison of how the mass changes over time.Examples provided in this patent where there is no deformation oftissues and only replacement of voxels include the following: insertionof a 3D digital object representing a radiofrequency ablation zone;insertion of a virtual coil into an aneurysm and insertion of femoralneck fixation hardware into the femur.

In accordance with an aspect an apparatus comprises: performingfiltering/segmentation, 3D cursor, 3D viewing via augmentedreality/virtual reality headset, manipulation of voxel(s), creation ofvoxel(s) and elimination of voxel(s) to optimize viewing of a structure.The example provided in this patent is of a cerebral arteriovenousmalformation (AVM). In this example, the feeding arteries, AVM nidus anddraining veins will be labeled. Any structures that block the viewing ofthe AVM will be either removed or made less conspicuous. The tightlypacked blood vessels will be spread apart through insertion of invisiblevoxels and tissue-type voxels. Through coordinated multi-voxelmanipulations, voxel creation and voxel elimination, the AVM can beuntangled (e.g., straightened, stretched, bent, twisted, expanded,contracted, cut, etc.) and the connections and AVM anatomy betterunderstood. Through this virtual radiological dissection and through theinsertion of dynamic voxels in the form of virtual contrast into an AVMaccompanied by varying virtual occluder insertion patterns, theunderstanding of the complex AVM anatomy can be enhanced in anon-invasive manner.

In accordance with an aspect a computer-readable medium comprises:instructions which, when executed by a computer, cause the computer tocarry out the steps of: assigning tissue type properties to voxels of amedical image; and performing a three-dimensional simulation bymanipulating the voxels based on the assigned tissue type properties andan input that prompts voxel manipulation. In some implementations thestep of assigning tissue type properties to the voxels of the medicalimage comprises assigning a value for at least one of: elasticity,ductility, hardness, density, and thermal conductivity. In someimplementations the step of manipulating the voxels comprises changingat least one of voxel size, voxel location, voxel orientation, voxelshape, voxel color, voxel grayscale, and voxel tissue type propertyvalue. In some implementations the input comprises inserting a virtualvolume-subtending surgical object. In some implementations the inputcomprises inserting a virtual volume-subtending anatomic object. In someimplementations the step of performing the simulation comprisesrepresenting virtual motion. In some implementations the step ofperforming the simulation comprises representing deformation of tissue.In some implementations the step of performing the simulation comprisesrepresenting a radiological dissection. In some implementations the stepof performing the simulation comprises creating new voxels in responseto the input. In some implementations the step of creating new voxelscomprises creating new fixed-type voxels that do not change unless actedupon by a user through an interface. In some implementations the step ofcreating new voxels comprises creating new invisible-type voxels. Insome implementations the step of creating new voxels comprises creatingnew tissue-type voxels. In some implementations the step of creating newvoxels comprises creating new dynamic-type voxels having at least onetissue type property that changes over time. In some implementations thestep of creating new voxels comprises creating new mobile voxels thattravel through a virtual vascular structure. In some implementations thestep of performing the simulation comprises eliminating selected voxelsin response to the input. In some implementations the instructionscomprise eliminating tissue type voxels to simulate ablation. In someimplementations the instructions comprise inserting strategicelimination points that designate a direction in which voxels areeliminated in discrete steps. In some implementations the instructionscomprise inserting strategic non-elimination points that designatevoxels that are preserved from elimination. In some implementations theinstructions comprise assignment of a strategic deformation feature thatdesignates a maximum shift for at least one voxel.

In some implementations, a method comprises: generating a 3D volumetricdataset through an artificial intelligence process; assigning at leastone mechanical type property to the 3D volumetric dataset; performingrendering of the 3D volumetric wherein the 3D volumetric dataset has afirst configuration; receiving an input to cause the 3D volumetricdataset to change from a to a subsequent configuration wherein thechange to a subsequent configuration is in accordance with the nature ofthe input and the mechanical type properties of the 3D volumetricdataset; changing the 3D volumetric dataset from a first configurationto the subsequent configuration; and performing rendering of the 3Dvolumetric dataset in the subsequent configuration.

The preferred method of generating a 3D volumetric dataset through anartificial intelligence process is through a generative adversarialnetwork. A generative adversarial network can generate and discriminatereal vs. fake data. Please see U.S. patent application Ser. No.16/703,629 titled Radiologist-assisted machine learning withinteractive, volume-subtending 3D cursor filed on 4 Dec. 2019 foradditional details on generating of a 3D volumetric dataset. Thepreferred embodiment is to generate the full set of data including everyinternal anatomic feature to the best possible detail (i.e., equal tothat of an actual scan), which will allow using this data for purposessuch as building training datasets for machine learning purposes indiagnostic radiology. An alternative embodiment, is to build onlyselected portions of the internal volumes using artificial intelligenceprocess. For example, if the rendering (see below) was only going to bedone of the skin surface, building the intricate internal volumes insidethe bone for example would not contribute to the actual rendering.However, building the fat, muscle, tendons, ligaments and outer surfacesof the bone would be useful for rendering purposes as realistic shapescould be formed. This embodiment reduces demand on CPUs and GPUs.

The preferred embodiment of assigning at least one mechanical typeproperty to the 3D volumetric dataset includes a voxel based assignment.Specifically, each voxel would be assigned a tissue type and each tissuetype would have a specific set of mechanical properties. This is usefulbecause voxels could be manipulated, as discussed above, and simulationscould be performed. For example, consider assigning a rigid property tothe mandible and a flexible property to the cheeks and lips. During awide opening of the mouth, the muscles contract, the rigid mandiblemoves downward and the cheeks and lips stretch to the new configuration.Thus, by assigning mechanical type properties to each voxel, simulationscan be performed.

The preferred embodiment of performing rendering is through the D3Dprocess, as first described in U.S. Pat. No. 8,384,771 Method andapparatus for three-dimensional viewing of images. In this disclosure, ahead display unit performs stereoscopic rendering and voxel filtering.This allows a user to see exactly the tissues of interest and eliminatethe tissues of non-interest. Note that this initial performing renderingof the 3D volumetric is wherein the 3D volumetric dataset has a firstconfiguration.

Next, the preferred embodiment of receiving an input to cause the 3Dvolumetric dataset to change from a to a subsequent configurationwherein the change to a subsequent configuration is in accordance withthe nature of the input and the mechanical type properties of the 3Dvolumetric dataset is discussed. The preferred embodiments includeintrinsic alterations (e.g., via muscular contraction, flow downpressure gradients, etc.). A first source of inputs of movement will bevia muscular contractions is discussed. Thus, each muscle will bemodeled with its origin at a first set of (x, y, z) coordinates, whichconnects to bone and its insertion at a second set of (x, y, z)coordinates which connects to a similar spot on a different portion ofbone or a different bone altogether. During a muscular contraction, themuscle fibers thicken and shorten. A single coordinate system could beperformed. For example, bone has its center coordinate with roll, pitchand yaw. Unless fractured, drilled or the like, bones would alwaysmaintain the same configuration. For example, muscles would have aorigin and insertion on the bones, but their configuration would changein relation to contractions. In addition, during muscular contractions,the position and orientation of the bones would also change. In someembodiments, a series of coordinate systems can be implemented toachieve and model complex movements. In contrast to a single mastercoordinate system (e.g., all voxels in the entire human body are plottedonto a single coordinate system), another embodiment is for onecoordinate system could be established for each bone since bones arerigid structures. For example, the scapula would have a first coordinatesystem. The humerus would have a second coordinate system. The radiuswould have a third coordinate system. And so on. For example, see62/939,685, Method and apparatus for development of an organ-specificcoordinate system for additional details. Other methods can be used forfluids (e.g., blood flow), which occurs down pressure gradients. Thealternative embodiment is to have inputs be from external sources. Someimplementation comprise wherein a force from a first 3D object causes asecond 3D object to deform. For example, a hard object can push on adeformable object, such that the hard object does not deform and thedeformable object does deform. For example, a virtual surgical devicecould be pressed against a soft tissue structure, which causes it todeform.

The preferred embodiment for changing the 3D volumetric dataset from afirst configuration to the subsequent configuration is to generate a newvoxelated dataset with a specific time step. For example, 60 datasetscould be generated per second (60 Hz) over a muscular contraction. Thepreferred embodiment is for a large number of datasets to be generated.Alternative embodiments would be to have a beginning pre-contractionimage, a mid-contraction image and a post-contraction image.

The preferred embodiment of performing rendering is through the D3Dprocess, as first described in U.S. Pat. No. 8,384,771 Method andapparatus for three-dimensional viewing of images. Note that renderingof fluids can also be performed as described in U.S. provisional patentNo. 62/906,125, A method and apparatus for stereoscopic rendering ofmobile fluids. Some implementation comprise generating virtual contrast.

Some implementation comprise performing voxel manipulation to achievethe subsequent configuration by changing at least one of voxel size,voxel location, voxel orientation, voxel shape, voxel color, voxelgrayscale, and voxel tissue type property value.

Some implementation comprise generating a 3D volumetric datasetcomprised of voxels with at least two different tissue type properties.Some embodiments comprise generating a 3D volumetric dataset comprisedof voxels with at least two different tissue type properties. Forexample, a simple model would be rigid bone structures and flexible softtissues. For example, these two types of structures together could builda simulated face. However, additional tissue type properties would beneeded for best modeling, to include muscles, fat, cartilage, etc. Otherexamples include bending of a knee, opening of a hand, and movements ofa face.

Some implementation comprise an artificial intelligence process thatuses a generative adversarial network. See U.S. patent application Ser.No. 16/703,629 for additional details.

Some implementation comprise performing advanced visualizationtechniques comprising at least one of rotation, zoom, filtering,segmentation, false color, prioritized volume rendering, and texture.Some embodiments comprise performing additional techniques, such asadvanced visualization, as described in: Some of the techniques in thispatent are performed by utilizing techniques described in: U.S. patentapplication Ser. No. 15/878,463, Interactive 3D cursor for use inmedical imaging; U.S. patent application Ser. No. 16/010,925,Interactive placement of a 3D digital representation of a surgicaldevice or anatomic feature into a 3D radiologic image for pre-operativeplanning; U.S. patent application Ser. No. 15/904,092, Processing 3Dmedical images to enhance visualization; U.S. patent application Ser.No. 15/949,202, Smart operating room equipped with smart surgicaldevices; U.S. Pat. No. 9,473,766, Method and apparatus for threedimensional viewing of images; U.S. Pat. No. 9,615,806, Method andapparatus for creation and display of artifact corrected threedimensional (3D) volumetric data from biplane fluoroscopic imageacquisition; U.S. patent Ser. No. 14/644,489, Method and apparatus forcreation and display of artifact corrected three dimensional (3D)volumetric data from biplane fluoroscopic image acquisition; U.S. Pat.No. 9,980,691, Method and apparatus for three dimensional viewing ofimages; U.S. Pat. No. 9,349,183, Method and apparatus for threedimensional viewing of images; U.S. patent application Ser. No.16/509,592, Implantable markers to aid surgical operations; U.S. patentapplication Ser. No. 16/524,275, Using geo-registered tools tomanipulate three-dimensional medical images; PCT/US19/478, A virtualtool kit for radiologists; U.S. patent application Ser. No. 16/594,139,Method and apparatus for performing 3D imaging examinations of astructure under different configurations and analyzing morphologicchanges; U.S. patent application Ser. No. 16/683,256, Method andapparatus for performing 3D imaging examinations of a structure underdifferent configurations and analyzing morphologic changes; U.S.provisional application No. 62/843,612, A method of creating acomputer-generated patient specific image; U.S. provisional applicationNo. 62/846,770, A method of prioritized volume rendering to improvevisualization of prioritized items within a 3D volume; U.S. provisionalapplication No. 62/850,002, A method of creating an artificialintelligence generated differential diagnosis and managementrecommendation tool boxes during medical personnel analysis andreporting; U.S. patent application Ser. No. 16/654,047, A method tomodify imaging protocols in real time through implementation ofartificial intelligence; U.S. provisional application No. 62/856,185, Amethod of image manipulation based on eye tracking; U.S. patentapplication Ser. No. 16/506,073, A method for illustrating direction ofblood flow via pointers; U.S. patent application No. 62/906,125, Amethod and apparatus for stereoscopic rendering of mobile fluids; and,U.S. patent application No. 62/939,685, Method and apparatus fordevelopment of an organ-specific coordinate system.

Some implementation comprise wherein the 3D dataset is a 3D object.Examples include any tangible object, such as bone, muscle, surgicalinstrument, tumor, or other non-medical related tangible objects.

Some implementation comprise creating at least two different 3D objects.Such objects can be a wide range of anatomic, surgical or other tangibledevices. Some implementation comprise wherein the at least two different3D objects are moved from a first position when the at least twodifferent 3D objects are separate to a second position wherein the atleast two different 3D objects are touching.

Some implementation comprise a time stepped interactive feature, as isdescribed in greater detail in U.S. patent application Ser. No.16/563,985, A method and apparatus for the interaction of virtual toolsand geo-registered tools. Some implementation comprise an educationalsimulation for surgical steps. Some implementation comprise interactionwith sound and tactile feedback.

Some implementation comprise wherein the 3D dataset has tissue typeproperties to mimic at least one pathology. For example, some cancers,such as breast cancers are hard. Lipomas are soft. DVTs are hard.Infections cause induration. Abscesses are fluctuant.

Some embodiments comprises an artificial intelligence process that use agenerative adversarial network. For example, techniques can be used,such as is described in U.S. patent application Ser. No. 16/703,629,Radiologist-assisted machine learning with volume-subtending 3D cursor.

Some embodiments comprise performing inputs to mimic muscularcontractions, including at least one of the group comprising: bending ofa knee; opening of a hand; and, movements of a face. For example,complex movements could achieve lifelike appearances moving in 4D, suchas a person talking or a person walking.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates an apparatus for surgical planning.

FIG. 2 illustrates voxel manipulation, creation, and eliminationprocesses that improve visualization of an anatomic structure.

FIG. 3 illustrates aspects of voxel manipulation in greater detail,including change in voxel size, shape, position, orientation, orinternal parameter.

FIG. 4 illustrates examples of a single voxel manipulation with changein voxel size, shape, position, orientation, or internal parameter.

FIGS. 5A, 5B, 5C, and 5D illustrate examples of coordinated multi-voxelmanipulation, including twisting, bending, straightening, and shrinking.

FIG. 6 is a flow diagram that illustrates segmentation of voxels intotissue types along with a look-up table of voxel parameters for specifictissue types.

FIG. 7 is a flow diagram that illustrates new voxel creation in greaterdetail, including fixed-type voxels, dynamic-type voxels andinteractive-type voxels.

FIGS. 8A, 8B, and 8C illustrate three examples of voxel manipulation andvoxel creation to separate closely spaced blood vessels or stretch bloodvessels as part of the untangling of an AVM.

FIG. 9 illustrates examples of insertion of dynamic-type voxels, whichin the illustrated example is virtual contrast flowing in a tubularshaped blood vessel with plug flow and laminar flow models.

FIG. 10 illustrates the altered virtual contrast flow through theplacement of an interactive voxel called a virtual occluder.

FIG. 11 illustrates an example of voxel manipulations (e.g., ischemicappearance of kidney) in relation to placement of interactive-voxels(e.g., virtual occluder placed over the renal artery).

FIG. 12 is a flow diagram that illustrates voxel elimination in greaterdetail, including volumetric-type elimination, surface-type elimination,discarding eliminated voxels, and placing eliminated voxels into avirtual specimen bucket for analysis.

FIG. 13 illustrates volumetric-type voxel elimination using a 3D cursorwhere voxels outside of the 3D cursor are eliminated and voxels insideof the 3D cursor are retained.

FIG. 14 illustrates voxel elimination via a volumetric-type eliminationwith the use of a 3D cursor where voxels inside of the 3D cursor areplaced into a virtual specimen container and the void is filled in bycoordinated multi-voxel shift of nearby voxels.

FIGS. 15A and 15B illustrate voxel elimination via a multi-stepsurface-type layer-by-layer approach, either via a portion of a surfaceat a time or a whole layer of the whole surface one voxel layer at atime.

FIGS. 16A, 16B, 16C and 16D illustrate the insertion into 3D medicalimaging datasets where distortion of the adjacent tissues is notperformed with four examples including radiofrequency ablation of arenal mass, coiling of an aneurysm, placement of femoral neck screws orcomparing a breast mass over multiple time points.

FIG. 17 illustrates coordinated multi-voxel manipulation to achievevirtual motion of anatomic structures within 3D medical imagingdatasets, such as motion at the knee joint.

FIG. 18 illustrates coordinated multi-voxel manipulation to achievevirtual motion of virtual objects inserted into 3D medical imagingdatasets, such as a motion of a virtual catheter inside of a bloodvessel with tunnel view shown.

FIG. 19 is a flow diagram that illustrates a fitting algorithm andstrategic deformation points, which can be used for placement of avirtual object into a 3D medical imaging dataset.

FIG. 20 illustrates an example of deformable tissues where a 3D digitalobject is inserted into volumetric medical images and there is partialnative voxel manipulation and partial native voxel elimination.

FIG. 21 illustrates an example of the use of deformable tissues andstrategic deformation points, such as placement of a breast implant intothe breast.

FIG. 22 illustrates an example of deformable tissues showing virtualexpected post-operative appearance with change over time, such asplacement of a nasal implant.

FIG. 23 illustrates a flow diagram for the use of voxel manipulationstrategies to improve understanding of complex 3D anatomy, such as acerebral arteriovenous malformation (AVM).

FIG. 24 illustrates a method for generating a 3D volumetric datasetthrough artificial intelligence processes and then performing alteringthe 3D volumetric dataset.

DETAILED DESCRIPTION

Some aspects, features and implementations described herein may includemachines such as computers, electronic components, optical components,and processes such as computer-implemented steps. It will be apparent tothose of ordinary skill in the art that the computer-implemented stepsmay be stored as computer-executable instructions on a non-transitorycomputer-readable medium. Furthermore, it will be understood by those ofordinary skill in the art that the computer-executable instructions maybe executed on a variety of tangible processor devices. For ease ofexposition, not every step, device or component that may be part of acomputer or data storage system is described herein. Those of ordinaryskill in the art will recognize such steps, devices and components inview of the teachings of the present disclosure and the knowledgegenerally available to those of ordinary skill in the art. Thecorresponding machines and processes are therefore enabled and withinthe scope of the disclosure.

The terminology used in this disclosure is intended to be interpretedbroadly within the limits of subject matter eligibility. The terms“logical” and “virtual” are used to refer to features that areabstractions of other features, e.g. and without limitation abstractionsof tangible features. The term “physical” is used to refer to tangiblefeatures. For example, multiple virtual computing devices could operatesimultaneously on one physical computing device. The term “logic” isused to refer to special purpose physical circuit elements and softwareinstructions that are stored on a non-transitory computer-readablemedium and implemented by multi-purpose tangible processors.

The following are incorporated by reference:

-   -   U.S. Provisional Patent Application 62/628,527 titled A METHOD        AND APPARATUS FOR INTERACTIVE PLACEMENT OF A DIGITAL        REPRESENTATION OF A SURGICAL DEVICE INTO RADIOLOGIC IMAGES,        filed Feb. 9, 2018;    -   U.S. Provisional Patent Application 62/695,868 titled        INTERACTIVE VOXEL MANIPULATION STRATEGIES IN VOLUMETRIC MEDICAL        IMAGING ENABLES VIRTUAL MOTION, DEFORMABLE TISSUE, AND VIRTUAL        RADIOLOGICAL DISSECTION, filed Jul. 10, 2018;    -   U.S. patent application Ser. No. 15/904,092 titled PROCESSING 3D        MEDICAL IMAGES TO ENHANCE VISUALIZATION, filed Feb. 23, 2018;    -   U.S. patent application Ser. No. 16/010,925 titled INTERACTIVE        PLACEMENT OF A 3D DIGITAL REPRESENTATION OF A SURGICAL DEVICE OR        ANATOMIC FEATURE INTO A 3D RADIOLOGIC IMAGE FOR PRE-OPERATIVE        PLANNING, filed Jun. 18, 2018, which is a non-provisional of        U.S. 62/628,527;    -   U.S. Pat. No. 8,384,771 titled METHOD AND APPARATUS FOR THREE        DIMENSIONAL VIEWING OF IMAGES; and    -   U.S. Pat. No. 9,980,691 titled METHOD AND APPARATUS FOR THREE        DIMENSIONAL VIEWING OF IMAGES.

U.S. 62/628,527 describes insertions of 3D digital objects intovolumetric medical images to improve pre-operative planning. The 3Ddigital objects are superimposed on the volumetric medical images, suchas where both an object and a portion of anatomy share the same imagecoordinates. Adjusting the transparencies of the object and anatomyenables both the anatomy and the inserted object to be contemporaneouslyvisualized. However, distortion of anatomic tissues resulting frominsertion of a volume-subtending digital surgical or anatomic object isnot represented by superimposition. It would therefore be useful to havea process to determine how to displace native voxels corresponding todisplacement of native tissues, including how those native voxels aredistorted when a new digital surgical or anatomic object is inserted.

Radiological imaging modalities including computed tomography (CT),magnetic resonance imaging (MRI), single photon emission computedtomography (SPECT), and positron emission tomography (PET) all acquirevolumetric medical imaging datasets and traditionally present thesedatasets in a slice-by-slice manner. Techniques from U.S. Pat. Nos.8,384,771, 9,980,691, U.S. patent application Ser. No. 15/904,092, andU.S. patent application Ser. No. 16/010,925, each of which isincorporated by reference, describe processes for building a volumetric3D dataset and enhanced viewing methods including a head display unit(HDU) such as an augmented reality (AR) or virtual reality (VR) headset,3D cursor, and other enhanced viewing methods. Aspects of theseprocesses may be combined, augmented, and improved upon to enablevirtual motion, deformable tissues, and virtual radiological dissectionthrough manipulation of voxels.

In some implementations virtual motion is achieved by assigningmaterial-like properties in the voxel manipulation process. Tissueproperties possibly including, but not limited to, mechanical propertiessuch as elasticity, ductility, hardness, and physical properties such asdensity and thermal conductivity, may be used to calculate displacementof native voxels corresponding to tissue distortion. Soft tissues can beassigned a deformable tissue property and bones can be assigned arigid-type tissue property, to match what naturally occurs. Anapplication of the process is representing tissue deformation duringplacement of a 3D virtual object into a 3D medical imaging dataset. Toguide the deformation process, strategic deformation points (or voxels)can be assigned.

Another limitation that exists in 3D medical imaging is the lack ofability to represent complex 3D anatomy due to proximity betweenmultiple branches of an anatomical feature. An example clinical scenariois a cerebral arteriovenous malformation where this limitation ofcomplex 3D anatomy and multiple vessels in close proximity maysignificantly impact patient care by hindering neurosurgical andneurointerventional radiology's decision making. If the precise anatomycould be better understood, then decision-making and treatment planningcould be improved, possibly along with improved outcomes. Voxelmanipulation, creation, and elimination can be utilized together toperform a virtual radiological dissection, for which the examplediscussed in detail herein is the untangling of a complexcerebrovascular arteriovenous malformation. A single voxel or group ofvoxels can be annotated to improve the radiological dissection process.Furth, the interactive voxel manipulation steps can be recorded andre-played from any angle to improve representation and understanding ofcomplex anatomy.

FIG. 1 illustrates an apparatus for implementing virtual motion,deformable tissues, and virtual radiological dissection throughmanipulation of voxels. A radiologic imaging system 200 (e.g., X-ray,ultrasound, CT (computed Tomography), PET (Positron EmissionTomography), or MM (Magnetic Resonance Imaging)) is used to generate 2Dmedical images 202 of an anatomic structure 204 of interest. The 2Dmedical images 202 are provided to an image processor 206, that includesprocessors 208 (e.g., CPUs and GPUs), volatile memory 210 (e.g., RAM),and non-volatile storage 212 (e.g. HDDs and SSDs). A program 214 runningon the image processor implements one or more steps as described below,e.g. and without limitation to generate simulations. 3D medical imagesare generated from the 2D medical images and displayed on an IO device216. The IO device may include a virtual or augmented reality headset,monitor, tablet computer, PDA (personal digital assistant), mobilephone, or any of a wide variety of devices, either alone or incombination. The IO device may include a touchscreen and may acceptinput from external devices (represented by 218) such as a keyboard,mouse, and any of a wide variety of equipment for receiving variousinputs. However, some or all the inputs could be automated, e.g. by theprogram 214.

FIG. 2 illustrates aspects of voxel processing by program 214 (FIG. 1).Input voxels of DICOM (Digital Imaging and Communications in Medicine)images are used to generate a 3D volumetric dataset as indicated inblock 200. The 3D volumetric dataset is characterized by an initialtotal voxel number and initial voxel properties, which may be assigned.The 3D volumetric dataset undergoes one or more of voxel manipulation202, voxel creation 204, and voxel elimination 206, in any combination,any order, and any number of iterations. The result of the processing isa processed 3D volumetric dataset 208, for which total voxel number andvoxel properties are altered relative to the input 3D volumetricdataset. Single voxel-manipulation and coordinated multi-voxelmanipulation in step 202 may include: alteration of voxel size;alteration of voxel location; alteration of voxel orientation;alteration of voxel shape; alteration of voxel internal parameters(e.g., gray-scale, color, tissue properties, mechanical properties,physical properties, etc.). Voxel creation in step 204 may include, butis not limited to, fixed-voxel creation (e.g., surgical-device-typevoxels, tissue-type voxels, invisible-type voxels, etc.), dynamic voxels(e.g., mobile voxels such as virtual contrast), and interactive voxels(e.g., strategic deformation points, strategic elimination points,virtual occluder, etc.). Voxel elimination in step 206 may include, butis not limited to, the following: an eliminate and discard-type approach(e.g., volume-type elimination, surface-type elimination), andelimination with placement of eliminated voxels into the virtualspecimen container bucket.

FIG. 3 illustrates aspects of the voxel manipulation process 202 ingreater detail. Aspects include one or more of size changes 300,position changes 302, orientation changes 304, internal parameterchanges 306, and shape changes 308, in any combination 310. Changingvoxel size 300 includes increasing or decreasing the notional volume ofa selected voxel. Changing voxel position 302 includes reconfiguring thelocation of the selected voxel, e.g. by changing the X, Y, Z coordinatescorresponding to a center point of the voxel. Changing voxel orientation304 includes altering one or more of the roll, pitch, and yaw of theselected voxel relative to a point of reference. Changing internalparameters 306 includes adjusting assigned values related tovisualization, e.g., color, gray-scale, texture, etc., values related tophysical properties, and values related to mechanical properties.Changing shape 308 includes altering the geometric shape of the selectedvoxel, e.g., cylinder, cone, sphere, cuboid, etc. The combination ofchanges converts the input set of voxel parameters 310 into an outputset of voxel parameters 312.

FIG. 4 illustrates aspects of manipulation of a single voxel in greaterdetail. Voxel manipulation is illustrated with respect to an input voxel400 having a cuboid shape. A manipulation 402 that decreases the size(volume) of the input voxel ratiometrically decreases the lengths ofedges of the voxel such that the shape of a resulting voxel 414 isscaled-down. A manipulation 404 that increases the size of the inputvoxel ratiometrically increases the lengths of edges of the voxel 400such that the shape of the resulting voxel 416 is scaled-up. Amanipulation 406 that alters the shape of the voxel 400 into a cylinderresults in a cylindrical voxel 418 characterized by the same volume,location, and orientation as the input voxel. A manipulation 408 thatalters the shape of the voxel into an octahedron results in a voxel 420characterized by the same volume, location, and orientation as the inputvoxel. A manipulation 410 of the center point of the voxel results inmovement from location 1 to location 2 within the 3D medical image. Amanipulation 412 of the orientation of the input voxel altersorientation while retaining shape, size and center point location,resulting in voxel 422. As already mentioned, the changes may beimplemented in combination, e.g. in series. Because the spatialresolution of many examinations (e.g., CT or MM) is 1 mm or smaller,small structures can have a poor aesthetic appearance. Take for example,a 3 mm blood vessel making a 180 degree turn over 1 cm. A voxeltransformation from cube-shape to cylindrical-shaped voxels wouldconstitute an improvement of visualization. It is anticipated thatperforming voxel transformations to improve visualization will directlyimprove diagnosis and patient care.

In the context of a 3D image, manipulation of a single voxel may promptmanipulation of adjacent voxels (unless there is empty space adjacent tothe changed voxel). For example, changes to one or more of size, shape,orientation, and location of the center point of the voxel may affectadjacent voxels. The way the adjacent voxels are affected is determinedat least in part by the internal parameters of the adjacent voxels.FIGS. 5A, 5B, 5C, and 5D illustrate examples of coordinated multi-voxelmanipulation, including twisting, bending, straightening, and shrinking.

Referring to FIG. 5A, virtual object twisting based on multi-voxelmanipulation can be used to improve visualization. For example, the usermay prompt twisting of a structure about its axis, such as performingroll, pitch, and yaw manipulations, or any combination thereof, or incombination with other voxel manipulations, voxel insertions, or voxelremovals. In the illustrated example a blood vessel 500 with a smallaneurysm 502 is rotated. As a result, the aneurism is relocated from topto bottom in the image. Assigned intrinsic tissue properties willdetermine the maximum amount of twist. During the twisting process, thecoordinates of the whole twisted vessel undergo a transformation inaccordance with the amount of twist. Additional twisting algorithms mayinclude, but are not limited to, the following: center of the vesseltwists a greater rotation than the off-center portions of the vessel;and, free ends (e.g., cut ends) of the vessel twists a greater rotationthan other portions of the vessels. In conjunction with other voxelmanipulations and insertions described herein, a virtual twistingprocedure may prove beneficial in improving understanding of a complexcerebral AVM (arteriovenous malformation).

FIG. 5B illustrates use of multi-voxel location and orientationmanipulation to improve visualization. In the illustrated example voxelmanipulation is used to generate a bend 503 in a straight blood vessel504. The bend is generated without stretching the feature beingmanipulated unless limited stretching is indicated by internalproperties. This may be useful for untangling features in a compleximage.

FIG. 5C illustrates another use of location and orientation manipulationto improve visualization. In the illustrated example voxel manipulationis used to straighten a non-linear section of a blood vessel 506,resulting in linear section 507. The change may be implemented withoutstretching the feature being manipulated unless limited stretching isindicated by internal properties. This may be useful for untanglingfeatures in a complex image.

FIG. 5D illustrates use of deformation and shrinking manipulations toimprove visualization of features of interest. In the illustratedexample, a gyms 508 of the brain is obscuring visualization of bloodvessels 510, 512. The user can, via the IO device, shrink or deform thegyms 508 such that the viewing of the blood vessels 510, 512 isoptimized. The voxels of the gyms are manipulated by shrinking of voxelsize and changing voxel location. The illustrated example depicts thegyms on end, disposed between the two blood vessels. The whole gyms(i.e. all layers) is shrunk such that the blood vessels can be bettervisualized depending on the viewing perspective. More particularly, thecluster of voxels associated with the gyms is deformed by changes insize, shape, and orientation of gyms voxels to account for the downwarddeformation, as are adjacent clusters of voxels. It should be noted thatalthough the shape, orientation, location, and size of the voxels may bechanged, voxels have not been subtracted. It will therefore beunderstood that deformation is not the same as subtraction. The resultof the illustrated tissue deformation is that the two parallel vesselsare in line of sight from one another, without the gyms being disposedtherebetween. A wide variety of voxel deformation manipulations arepossible including, but not limited to, the following: shrinking orexpanding a cluster of voxels based on their location; shrinking orexpanding a cluster of voxels based on by tissue type; and, altering thelocation of a portion of voxels, such as would be done if a rigid objectwas pushing onto the gyms of the brain. The ability to deform astructure helps to overcome an appreciable set of limitations in medicalimaging, which includes in the illustrated example the currentlimitation of one tissue type obscuring another.

To represent deformation of tissues, each tissue type in the body isassigned a tissue property of rigidness. Bone, for example, is highlyrigid, whereas brain tissue is moderately rigid, and unclotted blood isfluid with no internal rigidity. In some implementations, highly rigidstructures cannot be deformed, but may be altered via drilling methods.Structures of medium rigidity containing voxels within layers of thetissue could be deformed such that during the deformation the overallintegrity between the tissue layers remains unaltered, but the relativesize or number of voxels can be decreased in one area but increased inanother area. As an example, in gross anatomy it is possible to pushwith the finger downward to deform the shape of the gyms of brain suchthat it takes on a more flattened appearance but preserves the tissuelayers. Some implementations simulate such deformation.

FIG. 6 illustrates a process for assignment of voxel internalproperties. An image or portion thereof (e.g. volume of interest) issegmented into tissue types as indicated in step 600. In the illustratedexample the tissue types 602 are skin, fat, bone, cerebrospinal fluid,brain gray matter, and other tissues. Values associated with differenttypes of properties are then assigned to the voxels of each tissue typeas indicated in step 604. The values may be recorded in, and retrievedfrom, a look-up table 606. The illustrated look-up table includes valuesfor mechanical properties 608 and physical properties 610. Themechanical properties include elasticity, ductility, and hardness. Thephysical properties include density and thermal conductivity. Otherproperties, e.g. chemical properties, may also be included. Theparameter values are used to calculate voxel manipulations associatedwith tissue deformation, and thereby determine whether the representedtissues will deform in a realistic manner. As an example, skin should beassigned some intrinsic elasticity whereas bone should be assigned arigid-type property rather than an elastic-type property. As anotherexample, cortical bone should be assigned a high level of hardness andcartilage should be assigned a lower level of hardness.

FIG. 7 illustrates aspects of the voxel creation process. The first step700 is to determine which category of voxel to create. The categories ofvoxels include fixed-type voxels 702, dynamic-type voxels 704, andinteractive-type voxels 706. Fixed-type voxels do not change unlessacted upon by the user through the IO device. Fixed-type voxels may bepresent in the original DICOM data and may be created. Examples offixed-type voxels include, but are not limited to, the following:surgical-device-type voxels; tissue-type voxels; invisible-type voxels;common material-type voxels (e.g., water, air, etc.); and, many others.Dynamic-type voxels have at least one parameter that changes over timein a prescribed manner unless acted upon by the user. Dynamic-typevoxels do not exist in the original DICOM data but can be created by theuser. Examples of dynamic-type voxels include, but are not limited to,the following: virtual contrast where the position of the voxels changesover time to simulate blood flow; and, mobile objects, such as a 3Ddigital representation of a heart valve, which would open and close inaccordance with the heart beats. Interactive-type voxels can changefixed-type voxels and dynamic-type voxels in at least one way.Interactive-type voxels do not exist in the original DICOM data but canbe created. Examples of interactive-type voxels include, but are notlimited to, the following: virtual occluder which stops the movement ofvirtual contrast within a blood vessel; strategic deformation pointswhich helps guide the process of voxel manipulation (e.g., local tissuedistortion when inserting a 3D digital object); strategic eliminationpoints which helps to guide the elimination of voxels of non-interest;or others. After determining the category of voxel to create,corresponding parameters (e.g., size, shape, position, orientation,internal parameter, etc.) are assigned to the voxel as indicated in step708. The created voxels are then inserted into the medical imaging dataset as indicated at step 710. The insertion step includes manipulationof pre-existing voxels (i.e., those in place prior to the insertion).

FIG. 8A illustrates separation of the voxels of two closely-spacedfeatures 800, 802 via creation and insertion of invisible voxels 804. Asignificant limitation when viewing a complex 3D structure with multipleareas of tight overlap is the inability to visualize and assess deeperlayers that are obscured. Spreading features apart can improvevisualization from all angles by reducing overlap. One technique forspreading features apart is to create invisible (clear) voxels that areinserted between the features. The invisible voxels affect the voxelsassociated with the features, e.g. causing a change in position of oneor both features.

Separation of the fragments of blood vessels may also be accomplished byapplying an additive factor to the coordinates of a feature. In theillustrated example the features 800, 802 are two fragments of bloodvessels. For context, the two blood vessels may have diameters of 10voxels and be separated by 10 voxels. The top row of the top bloodvessel has coordinates (99, 100, 100), (100, 100, 100), (101, 100, 100),and (102, 100, 100). The bottom row of the top blood vessel hascoordinates (99, 90, 100), (100, 90, 100), (101, 90, 100), and (102, 90,100). The top row of the bottom blood vessel has coordinates (99, 80,100), (100, 80, 100), (101, 80, 100), and (102, 80, 100). The bottom rowof the bottom blood vessel has coordinates (99, 70, 100), (100, 70,100), (101, 70, 100) and (102, 70, 100). The coordinates of the bottomvessel may remain unchanged while an additive factor of 100 is appliedto the y-values of the top vessel. The result is a shift in relativepositions of the blood vessel fragments, resulting in an increase inseparation at a magnitude determined by the value of the additivefactor. The final coordinates of the top row of the top vessel in theillustrated example are (99, 200, 100), (100, 200, 100), (101, 200,100), and (102, 200, 100). The final coordinates of the bottom row ofthe top vessel is (99, 190, 100), (100, 190, 100), (101, 190, 100) and(102, 190, 100). Invisible voxels may be created and inserted into thespace created by the process.

Voxel counting metrics 806 may also be used. The voxels of a featurecould be selected to have surfaces that better portray the feature. Forexample, the distance along the surface of blood vessel may be moreaccurately measured if the blood vessel is represented bycylindrical-type voxels as opposed to cuboids at even spacing. The voxellengths along a curvilinear edge are added together to generatesub-total or total lengths. In the illustrated example, voxel countingmetrics for a curvilinear distance are illustrated. Assume, for example,the length of the inferior most aspect of a straight blood vessel hascoordinates (0, 0, 0), (1, 0, 0), (2, 0, 0) and (3, 0, 0); the distanceof this portion of the blood vessel is 3 voxels. However, if the bloodvessel is curving slightly with the length of the bottom surface of theblood vessel having coordinates of (0, 0, 0), (1, 0, 1), (2, 0, 2) and(3, 0, 3); using Pythagorean's theorem, the distance would be 3√2voxels. A key advantage the voxel counting metrics is the enhancedpre-operative planning resulting from a better understanding of thedistance that a catheter needs to move within a blood vessel to reach adesired position to perform a desired task. Other counting metrics maybe associated with surface areas and volumes.

FIG. 8B illustrates tissue stretching via creation and insertion of bothinvisible voxels 808 and tissue-type voxels 810. Tissue stretching mayhelp to overcome the challenge of overlapping tissues. A strategic cutpoint 812 on the feature 814 is selected. The feature is segmented atthe strategic cut point, resulting in segments 816, 818 and stretchpoints 820, 822. One of the segments is pulled away from the othersegment, e.g. segment 818 pulled-away from segment 816. This may beaccomplished by inserting invisible voxels 808 or by applying anadditive factor to the coordinates of the voxels of the segment that isbeing moved. To facilitate visualization of segment interconnection,computer generated lines 824 that connect the two previously contiguous,but now geographically separated, segments 816, 818 may be generated.The lines could be displayed in some enhanced fashion (e.g., falsecolor, dashed, blinking, thin, etc.) to denote to the user which vesselis native and which vessel is computer generated. As in the previouslydescribed example, one of the segments may remain stationary while theother segment is relocated. Tissue-type voxels 810 are created andinserted to interconnect the two segments, e.g., between the lines 824and bridging the corresponding stretch points 820, 822.

FIG. 8C illustrates use of tissue stretching to separate two closelyspaced blood vessel branches 825, 827 that share a Y-shaped fork, suchas commonly occurs in cerebrovascular AVMs. Strategic cut points 826,828 from which to stretch each branch outward are selected. At eachstrategic cut point the corresponding branch is pulled away, e.g. byapplying an additive factor. The branches may be assigned differentadditive factor values. Stretch points 830, 832, 834, 836 are therebycreated. To help visualization of how the created segments interconnect,computer-generated lines 835 connect the two previously contiguous, butnow geographically separated, segments on each branch of the nativelyclosely spaced forked vessel. The lines could be displayed in someenhanced fashion (e.g., false color, dashed, blinking, thin, etc.) todenote to the user which segment is native and which segment iscomputer-generated. Tissue-type voxels 837 are created and inserted tointerconnect the two segments, e.g., between the lines and bridging thecorresponding stretch points. Further, invisible voxels 838 can becreated and inserted to fill the void between the newly geographicallyseparated segments. Thus, a user can effectively pull-apart a complexstructure (e.g., cerebral AVM) such that the tangle and connections canbe better visualized and understood.

FIG. 9 illustrates insertion of dynamic-type voxels 900 to representvirtual contrast flowing in an artery 901 with plug flow 902 and laminarflow 904 models. Typically, this injection would occur in an artery, butwould not necessarily be restricted to the artery. It could also beperformed in a vein or even GI track lumen. At time point #1, there isno virtual contrast within the artery. At time points #2, #3, and #4,the virtual contrast progressively fills in from proximal to distal. Thevirtual laminar flow model 904 has faster flow in the center of theartery as compared to the lumen closest to the wall. A wide variety ofother flow models can be implemented, such as accounting for normalblood vessels or diseased blood vessels (e.g., stenosis due toatherosclerosis).

FIG. 10 illustrates altered virtual contrast flow resulting fromplacement of an interactive voxel 101 such as a virtual occluder. Forcomparison, both normal virtual contrast flow and altered virtualcontrast flow with placement of a virtual occlude are shown. The virtualcontrast 103 can progress from proximal to distal in the virtuallyoccluded branch 105 up to the point of the virtual occluder, but notbeyond the virtual occluder. Insertion of virtual contrast is notimpeded in the non-occluded branches. Thus, the insertion of aninteractive voxel, such as a virtual occluder, can simulate placement ofa surgical clip.

FIG. 11 illustrates automatic voxel manipulations in response toplacement of interactive-voxels. In the illustrated example the ischemicrepresentation of a kidney changes in response to placement of virtualoccluders 111 over the renal artery 113A. 113B illustrates the renalvein. 113C illustrates the ureter. The kidney changes from the normalpink color to blue as the blood supply is cut off. Voxel alteration toreflect pathologic process of renal ischemia. More specifically, thekidney slightly decreases in size and changes in color (or grayscalebased on user preferences). Voxel manipulation to reflect expectedchanges in blood flow after the interactive virtual occluder is placed.The voxels of the kidney parenchyma are interactive-type voxels that arepink when well perfused but change to blue when the blood supply is cutoff by the virtual occluder. Virtual representation of such a proceduremight be implemented where an individual is undergoing a nephrectomy forrenal transplant or renal cell carcinoma. It is anticipated that theinsertion of interactive voxels will aid in diagnostic accuracy andsurgical planning.

FIG. 12 illustrates voxel elimination. Step 121 is to select a voxelelimination strategy type. Examples include volumetric-type eliminationand surface-type elimination. A volumetric-type elimination can beperformed using a 3D cursor such as that described in U.S. Pat. No.9,980,691 to select a volume. An example of this would be to place thevolume-subtending 3D cursor over the area of interest and subtract allvoxels outside (or inside) the volume. Surface-type elimination removesa single-voxel deep layer of the whole surface, which would allowanalysis of a tumor or ablate away areas of non-interest that are in theway. Alternatively, the insertion of a virtual object could override(and replace) native voxels for elimination. The process may be guidedthrough placement of strategic elimination points and strategicnon-elimination points. Eliminated voxels can be either discarded asshown in step 123 or placed into a virtual specimen bucket for analysisas shown in step 125. After the elimination and removal of voxels asindicated in step 127, manipulation of the pre-existing voxels (i.e.,those in place prior to the elimination) could be performed to fill thevoid created as indicated in step 129.

FIG. 13 illustrates an example of voxel elimination via volumetric-typeelimination with use of a 3D cursor 131. Voxels 133 located outside ofthe volume defined by the 3D cursor are eliminated. Voxels 135 locatedinside of the volume defined by the 3D cursor are retained. This voxelelimination technique may be used to isolate an area of interest, suchas a tumor and the immediately surrounding anatomy.

FIG. 14 illustrates voxel elimination via a volumetric-type eliminationwith the use of a 3D cursor 141. Voxels located inside of the 3D cursorin image 143 are placed into a virtual specimen container 145. The void147 resulting from voxel elimination, as shown in image 151, is filledin by coordinated multi-voxel shift of nearby voxels 149, as shown inimage 153. The virtual specimen container can be used for many purposes.If the structure is unknown, it can be analyzed pre-operatively incomparison with other similar tissues in an attempt to refine thedifferential diagnosis, so that the medical team would be moreconfident. After getting a definitive diagnosis via pathology, theobject could be labeled and added to a training database for machinelearning purposes. After the object is removed, an option would be formanipulation of local voxels, such as filling in with the adjacent fattissue.

FIG. 15A illustrates virtual ablation of a portion of a layer of thesurface of a gyms 159 of brain using voxel elimination via a multi-stepsurface-type layer-by-layer approach. This may be useful where the gymsis disposed between blood vessels 155, 157. Strategic elimination points151 (or strategic elimination voxels) are designated in initial image153. These points can be placed underneath the desired ablation surfaceto guide the direction of the surface ablation. Voxels are removed,layer-by-layer, in a directed fashion, such that smaller and smallersurface shells of a structure are generated as shown in progression byimages 161, 163, and 165. Image 161 illustrates the outer most shell ofgyms ablated improving visualization of the vessels. Image 163illustrates the next shell of gyms ablated improving visualization ofthe vessels. Image 165 illustrates the gyms of brain obscuresvisualization of the vessels. Initially, a row 167 of the top layer ofthe cortex may have voxels (100, 100, 100), (101, 100, 100), (102, 100,100) and (103, 100, 100). Upon removal of this top layer of cortex, thenew top row 169 would then be (100, 99, 100), (101, 99, 100), (102, 99,102) and (103, 99, 100). Upon removal of that layer of cortex, the nexthighest row 171 would be (100, 98, 100), (101, 98, 100), (102, 98, 102)and (103, 98, 100). The eliminated voxels are removed from the displayedvolume.

FIG. 15B illustrates virtual ablation of a whole layer of the surface ofa mass 175 using strategic non-elimination points 173 (or strategicnon-elimination voxels). These can be placed within the desiredstructure that will be ablated to designate the no further ablationzone. Images 179, 181, and 183 illustrate virtual ablation of shellsfrom outer to inner relative to pre-ablation image 177. Image 179illustrates the outer most shell of the mass ablated. Image 181illustrates the next shell of the mass ablated. Image 183 illustratesthe next shell of the mass ablated. Uses for surface-type layer-by-layerablation may include, but are not limited to, the following: removing aportion of an object of non-interest while preserving some of it forcontext; simulating the effects of chemotherapy; simulating the effectsof radiation therapy; and, viewing the inside of a tumor in alayer-by-layer fashion. Some of the structures are less responsive tochemotherapy and these regions could be assigned a voxel propertydesignated as more difficult to ablate.

FIGS. 16A, 16B, 16C, and 16D illustrate specific examples of virtualinsertion into 3D medical imaging datasets where distortion of theadjacent tissues is not performed. The examples include radiofrequencyablation of a renal mass, coiling of an aneurysm, placement of femoralneck screws, and comparing a breast mass over multiple time points, eachof which is simulated using the techniques described above.

FIG. 16A illustrates a renal mass 191, for which a treatment calledradiofrequency ablation can be performed. The radiofrequency ablationtreatment is simulated by inserting a virtual ablation zone in image 193using techniques described above. Image 195 shows a computer-generated3D digital object 197 that is placed within the kidney 199, replacingvoxels that correspond to the mass 191 and that correspond to somenormal kidney tissue. Insertion includes only the replacement of normalkidney voxels and kidney mass voxels with virtual ablation zone voxels.No adjacent voxels (or tissues) are distorted.

FIG. 16B illustrates a blood vessel 201 with an aneurysm 203, for whicha treatment called endovascular coiling can be performed. Theendovascular coiling is simulated by inserting a virtual coil 205 intothe aneurysm 203 in image 207 using techniques described above.Resulting image 209 shows the virtual coil as a computer-generated 3Ddigital object that is placed within the aneurysm replacing voxels thatcorrespond to the blood inside the aneurysm sac. The insertion includesonly the replacement of blood voxels with virtual coil mass voxel. Noadjacent voxels (or tissues) have been distorted.

FIG. 16C illustrates a femur 211, for which placement of a femoralprosthesis can be performed in the event that the femur is fractured.The procedure is simulated by inserting virtual femoral neck fixationhardware 213 in image 215. Resulting image 217 shows the hardware as acomputer-generated 3D digital object that is placed within the femur andreplaces other voxels such as bone or bone marrow. The insertionincludes only the replacement of cortical bone voxels and bone marrowvoxels with virtual femoral neck fixation voxels. No adjacent voxels (ortissues) have been distorted.

FIG. 16D illustrates superimposition 223 of temporal versions 219, 221of a segmented-out breast mass taken at different points in time. Thebreast is a mobile structure and can change positions, but can alsochange in configuration (e.g., flatten, etc). Because of this, accuratecomparison of the mass at two points in time is difficult to represent.To properly register and compare the two temporal versions 219, 221 ofthe mass, one or both of the temporal versions may need to be rotatedand/or deformed (since the breast is a soft tissue). Performingdeformation of the breast masses such that they are superimposed, havethe same orientation, and are deformed the same way using the techniquesdescribed above may prove to help improve comparison of how the breastmass changes over time. Note that the superimposition represents thesame mass at different time points when it is different sizes. Properregistration of the masses is essential in order for precise comparisonof the size, spiculations, margins, involvement of adjacent structures,etc. Thus, rotation, translation, and deformation of the masses may berequired to achieve best comparison.

FIG. 17 illustrates coordinated multi-voxel manipulation to achievevirtual motion of anatomic structures within 3D medical imagingdatasets. Virtual motion is implemented by allocating tissue propertiesto each tissue within a volume and simulating movement of at least oneanatomic structure. The anatomic structure in the illustrated example isa knee joint. Image shows three bones (femur 225, patella 227, tibia229) at the knee joint. By assigning each voxel in these bones arigid-type tissue property, motion at the knee joint can be modeled andtranslated in position via a coordinated multi-voxel shift using thetechiques described above. All three bones shift in orientation andposition from image 231 to image 233. This requires a coordinatedmulti-voxel shift. For simplicity, soft tissues are not shown. Note thatno voxels are eliminated or created. Voxels are only manipulated. Thebones have rigid-type properties and therefore would be non-deformableif pushed upon. However, since the bones meet at a joint, virtual rangeof motion can be performed. The virtual motion can be viewed in 3D usingaugmented reality headset, e.g. as described in U.S. Pat. No. 8,384,771.

FIG. 18 illustrates coordinated multi-voxel manipulation to achievevirtual motion of virtual objects inserted into 3D medical imagingdatasets. The specifically illustrated example is motion of a virtualcatheter 235 inside of a blood vessel 237, rendered in both side view239 and tunnel view 241 at three different points in time. This exampleillustrates use of voxel manipulation to represent real-time movement ofanatomic structures or virtual objects within a 3D medical imagingdataset. The virtual surgical object can take on many forms, one ofwhich is a vascular catheter. The virtual object, e.g. virtual catheter,is assigned physical properties (e.g., hardness, malleability, etc.), ina manner like that of tissues as already described above. Thus, when thesoft virtual catheter is pushed through the blood vessel, the virtualcatheter deforms when it is pressed against a more rigid blood vessel.

The tip of the virtual catheter 235 is bent, which enables efficientnavigation through the branches of the blood vessel. Assume, forexample, that the first several coordinates of the top row of a bloodvessel 237 of diameter of 20 voxels were (100, 100, 100), (101, 100,100), (102, 100, 100), (103, 100, 100), (104, 100, 100), and (105, 100,100). And, assume that the first several coordinates of the bottom rowof that blood vessel were (100, 80, 100), (101, 80, 100), (102, 80,100), (103, 80, 100), (104, 80, 100) and (105, 80, 100). The distalportion of a vascular catheter commonly used to track along the insideof a blood vessel could take on the coordinates (100, 90, 100), (101,90, 100), (102, 90, 100), (103, 91, 100), and (104, 92, 104). If thecatheter is advanced one voxel in the x-direction to reach anx-coordinate of 105, it would take on the coordinates (100, 90, 100),(101, 90, 100), (102, 90, 100), (103, 90, 100), (104, 91, 104), and(105, 92, 100). The bent tip of the catheter accounts for the deviationin the y-direction. Because the material inside the vessel is liquidblood, the material is free to move and remaining tissues will not needto undergo virtual tissue deformation. The tunnel view 241 can be fromthe perspective of the distal catheter looking forward from the longportion of the catheter connected to the tip. The catheter tip and theupcoming vascular branches can serve to aid in both the navigationprocess and the understanding of the vascular lumen.

FIG. 19 is a flow diagram that illustrates a fitting algorithm andstrategic deformation points that can be used for placement of a virtualobject into a 3D medical imaging dataset. After tissue-type voxels andvirtual surgical object-type voxels are assigned tissue properties, andstrategic deformation points are assigned, as indicated in step 251, theuser specifies the desired placement location and surgical path by whichthe placement occurs. The algorithm determines if the placement ispossible as indicated in step 253. Object placement may be performed viaone of three options: coordinated multi-voxel shift of native tissuevoxels 257; partial replacement of native voxels and partial coordinatedmulti-voxel shift of native voxels 259; and complete replacement ofnative breast tissue voxels with 3D digital object voxels 261. It isimportant in at least some implementations to have the interactivecomponent of use of the strategic deformation points to achieve desiredfitting. Step 257 may be selected in the case where the preset fittingrules are satisfied. An example 263 is insertion of breast implantvoxels with coordinated multi-voxel shift of native breast tissuevoxels. An option 265 is to add predicted post-operative appearance atmultiple time points (e.g., edema seen at 1-day post-op, but not at1-year post-op). In the case where the preset fitting rules are notsatisfied the algorithm provides feedback to the user as indicated instep 255. Steps 259 and 261 are options in that case. An example 265 inwhich step 259 is selected is insertion of breast implant voxels withpartial native breast tissue voxel shift and partial native breasttissue voxel elimination. An example 267 in which step 261 is selectedis insertion of breast implant with replacement of native breast tissuevoxels with 3D digital object voxels.

FIG. 20 illustrates an example of virtual deformable tissues where a 3Ddigital object is inserted into volumetric medical images and there ispartial native voxel manipulation and partial native voxel elimination.The specifically illustrated example is insertion of virtual renaltransplant. The insertion includes the addition of voxels from anotherpatient's 3D medical imaging exam (from the donor kidney 269) onto thecurrent patient's 3D medical imaging exam (the recipient of the renaltransplant); note that the larger donor kidney distorts the appearanceof the recipient's native adrenal gland 271 located above the recipientkidney 273, which is achieved through a coordinated multi-voxel shift.

FIG. 21 illustrates an example of the use of deformable tissues andstrategic deformation limiting features to simulate placement of abreast implant 275 into the breast 277. The strategic deformationlimiting features help guide the insertion process. In this example,strategic deformation points 279 are placed near the top and bottom ofthe field of view 283, 285, and a strategic deformation line 281 isplaced at the back of the breast where the ribs are located. Suchstrategic deformation limiting features help set maximum shift valuesand are helpful at the edge of the field of view. The strategicdeformation points may be assigned a 1 cm maximum motion limit, and thestrategic deformation line may be assigned a 1 mm maximum motion limit,for example, and without limitation. Note that although the strategicdeformation limiting features have minimal to no shift, the remainder ofthe breast undergoes a coordinated multi-voxel breast tissue shiftduring placement of the 3D digital representation of the breast implant275. Maximum motion limits may be assigned to other points at the edgeof the field of view. For example, in an MM of the breast, the user caninsert a strategic minimal deformation point at the skin below thebreast at the inferior most aspect of the field of view and then selectfor the maximum amount of shift that this point can be stretched.

FIG. 22 illustrates use of virtual deformable tissues to simulateexpected post-operative appearance with change over time. Thespecifically illustrated example is placement of a nasal implant 301.The simulation is achieved by creating and inserting both fixed-type anddynamic-type voxels in a pre-operative image 303. Specifically, theedema 305 is represented by a dynamic-type voxel, which is present onthe simulated post-operative day #1, but resolves completely by thesimulated post-operative day #30. Alternatively, virtual disappearancecan be performed (e.g., certain tissues, such as fat-graft placementsare known to shrink over time; other conditions such as edema in thesoft tissues, volume overload in the vasculature are also known to betemporary and can undergo part or total disappearance over time).

FIG. 23 illustrates use of voxel manipulation processes to improveunderstanding of complex 3D anatomy. The specifically illustratedexample is a cerebral arteriovenous malformation (AVM) 351. Multipleaspects described above are used to facilitate pre-operative planning,including determining how to best treat the cerebrovascular AVM. Theexample AVM is characterized by three feeding arteries, a complex tangleof blood vessels, an intranidal aneurysm within the tangle (blackcircle) and four draining veins. It is difficult to understand from anunprocessed image how each of the branches of the AVM connect. Treatmentoptions include open surgical resection including clip placement overcertain blood vessels or endovascular embolization. A virtualradiological dissection 353 is performed to improve understanding of thecomplex 3D anatomy. A virtual occluder is then inserted as shown in step355, followed by insertion of virtual contrast as shown in step 357. Thevasculature is then re-assessed as indicated in step 359. The steps maybe iterated, and optionally recorded.

The virtual radiological dissection step 353 may include a variety ofsub-steps to untangle the complex structure of the AVM such that each ofthe branches can be better understood. For example, the feedingarteries, AVM nidus and draining veins may be labelled as indicated insub-step 361. Viewing may be enhanced or optimized with filtering,segmentation, 3D cursor use, 3D headset viewing. Any structures thatblock the viewing may be ablated or deformed to minimize obscuration ofthe AVM as indicated in sub-step 363. Creation and insertion ofinvisible-type voxels and tissue-type voxels connects the previouslycontiguous, but now separated structures as indicated in sub-step 365.Next, a series of untangling processes possibly including one or more ofstraightening, stretching, bending, and twisting are performed asindicated in sub-step 367. Virtual cutting may be performed as indicatedin sub-step 369.

A variety of features, aspects, embodiments and implementations havebeen described. Nevertheless, it will be understood that a wide varietyof modifications and combinations may be made without departing from thescope of the inventive concepts described herein. Accordingly, thosemodifications and combinations are within the scope of the followingclaims.

FIG. 24 illustrates a method for generating a 3D volumetric datasetthrough artificial intelligence processes and then performing alteringthe 3D volumetric dataset. Step 2400 is to generate a 3D dataset via AIprocesses, such as is performed in U.S. patent application Ser. No.16/703,629. Step 2401 is to assign at least one mechanical type property(e.g., tissue type property) to the 3D volumetric dataset. Step 2402 isto perform rendering of the 3D volumetric dataset wherein the 3Dvolumetric dataset has a first configuration. Step 2403 is to receive aninput to cause the 3D volumetric dataset to change to a subsequentconfiguration wherein the change to a subsequent configuration is inaccordance with the nature of the input and the mechanical typeproperties of the 3D dataset. Step 2404 is to change the 3D volumetricdataset to the subsequent configuration (e.g., through voxelmanipulation). Step 2405 is to perform rendering of the 3D volumetricdataset in the subsequent configuration.

What is claimed is:
 1. A method comprising: applying an artificialintelligence process to a 3D volumetric dataset wherein said 3Dvolumetric dataset contains discrete structures; performing segmentationof said 3D volumetric dataset to generate segmented structures whichcorrespond to said discrete structures; assigning properties to voxelsof said 3D volumetric dataset based on said segmented structures whereinsaid properties are determined by said discrete structures; presenting,at a first time step, said 3D volumetric dataset to a user; performing adeformation of said 3D volumetric dataset based on said propertieswherein said deformation changes at least one of the group consisting ofa size of a segmented structure, a shape of a segmented structure, alocation of a segmented structure, and an orientation of a segmentedstructure to generate a modified 3D volumetric dataset; and presenting,at a second time step, said modified 3D volumetric dataset to said user.2. The method of claim 1 further comprising wherein said 3D volumetricdataset mimics at least one of the group of a CT scan, a MM scan, a PETscan, a SPECT scan, and an ultrasound.
 3. The method of claim 2 furthercomprising performing voxel manipulation to achieve said deformation bychanging at least one of voxel size, voxel location, voxel orientation,voxel shape, voxel color, voxel grayscale, and voxel tissue typeproperty value.
 4. The method of claim 1 further comprising wherein said3D volumetric dataset is comprised of voxels with at least two differenttissue type properties.
 5. The method of claim 1 further comprisinggenerating virtual contrast.
 6. The method of claim 1 further comprisingwherein said artificial intelligence process comprises a generativeadversarial network.
 7. The method of claim 1 further comprisesperforming advanced visualization techniques comprising at least one ofrotation, zoom, filtering, segmentation, false color, and prioritizedvolume rendering.
 8. The method of claim 1 further comprising whereinsaid 3D volumetric dataset is a 3D object.
 9. The method of claim 8further comprising creating at least two different 3D objects.
 10. Themethod of claim 9 further comprising wherein the at least two different3D objects are moved from a first position wherein the at least twodifferent 3D objects are separate to a second position wherein the atleast two different 3D objects are touching.
 11. The method of claim 10further comprising wherein a force from a first 3D object causes asecond 3D object to deform.
 12. The method of claim 11 furthercomprising a time stepped interactive feature.
 13. The method of claim 1further comprising interaction with sound and tactile feedback.
 14. Themethod of claim 1 further comprising an educational simulation forsurgical steps.
 15. The method of claim 1 further comprising whereinsaid properties mimic at least one pathology.
 16. The method of claim 1further comprising generating blood flow appearance.
 17. The method ofclaim 1 further comprising utilizing at least two different tissue typeproperties to mimic muscular contractions, including at least one of thegroup of bending of a knee, opening of a hand, and movements of a face.18. An apparatus comprising: an TO device; and an image processor incommunication with the IO device, the image processors comprising aprogram stored on a computer-readable non-transitory media, the programcomprising instructions that perform: a step for applying an artificialintelligence process to a 3D volumetric dataset wherein said 3Dvolumetric dataset contains discrete structures; a step for performingsegmentation of said 3D volumetric dataset to generate segmentedstructures which correspond to said discrete structures; a step forassigning properties to voxels of said 3D volumetric dataset based onsaid segmented structures wherein said properties are determined by saiddiscrete structures; a step for presenting, at a first time step, said3D volumetric dataset to a user; a step for performing a deformation ofsaid 3D volumetric dataset based on said properties wherein saiddeformation changes at least one of the group consisting of a size of asegmented structure, a shape of a segmented structure, a location of asegmented structure, and an orientation of a segmented structure togenerate a modified 3D volumetric dataset; and a step for presenting, ata second time step, said modified 3D volumetric dataset to said user.19. The apparatus of claim 18 further comprising wherein said 3Dvolumetric dataset mimics at least one of the group of a CT scan, a MRIscan, a PET scan, a SPECT scan, and an ultrasound.
 20. A methodcomprising: a step for applying an artificial intelligence process to a3D volumetric dataset wherein said 3D volumetric dataset containsdiscrete structures; a step for performing segmentation of said 3Dvolumetric dataset to generate segmented structures which correspond tosaid discrete structures; a step for assigning properties to voxels ofsaid 3D volumetric dataset based on said segmented structures whereinsaid properties are determined by the discrete structures; a step forpresenting, at a first time step, said 3D volumetric dataset to a user;a step for performing a deformation of said 3D volumetric dataset basedon said properties wherein said deformation changes at least one of thegroup consisting of a size of a segmented structure, a shape of asegmented structure, a location of a segmented structure, and anorientation of a segmented structure to generate a modified 3Dvolumetric dataset; and a step for presenting, at a second time step,said modified 3D volumetric dataset to said user.